How to change the number of error retries in AWS SDK for Python (Boto3)
Although API calls may temporarily fail due to various factors, each AWS SDK implements automatic retries. In this blog, we will introduce how to change the number of error retries in the AWS SDK for Python (Boto3).
Please refer to the AWS Documentation about error retry and exponential backoff. - Error Retries and Exponential Backoff in AWS
Error Retries and Exponential Backoff in Boto3
Although retry processing is implemented automatically, the AWS SDK for Python (Boto3) seems to retry five times by default with exponential backoff.
Boto 3 Documentation | Config Reference
'max_attempts' -- An integer representing the maximum number of retry attempts that will be made on a single request. For example, setting this value to 2 will result in the request being retried at most two times after the initial request. Setting this value to 0 will result in no retries ever being attempted on the initial request. If not provided, the number of retries will default to whatever is modeled, which is typically four retries.
"retry": {
"__default__": {
"max_attempts": 5,
"delay": {
"type": "exponential",
"base": "rand",
"growth_factor": 2
},
Change the error number of retries
We are able to change the error number of retry it by setting retries config (botocore.client.Config) when creating session or service clients.
Set retries to config when creating session
ex.) Set the error retries up to 10 times for API call
from boto3.session import Session
from botocore.client import Config
session = Session()
config = Config(
retries=dict(
max_attempts=10
)
)client = session.client('ec2', config=config)
Set retries to config when creating client
ex.) Set the error retries up to 10 times for API call
import boto3
from botocore.config import Config
config = Config(
retries=dict(
max_attempts=10
)
)
client = boto3.client('ec2', config=config)
Conclusion
Exponential backoff is an essential algorithm in retries. This time we introduced the concept with the AWS SDK for Python (Boto 3). However, we would recommend implementing retry logic incorporating exponential backoff even when you are not using the AWS SDK. In Python, there is a module called retrying.
